Skip to content

Feat: add bounded whole-run FIFO admission - #1541

Open
Crane-Liu wants to merge 1 commit into
hw-native-sys:mainfrom
Crane-Liu:codex/worker-async-b2-whole-run-fifo
Open

Feat: add bounded whole-run FIFO admission#1541
Crane-Liu wants to merge 1 commit into
hw-native-sys:mainfrom
Crane-Liu:codex/worker-async-b2-whole-run-fifo

Conversation

@Crane-Liu

@Crane-Liu Crane-Liu commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Reserve a generation-safe pipeline lease before graph construction and admit at most one active plus one prepared whole run.
  • Partition SUB and NEXT_LEVEL ready queues by run; the scheduler dispatches only the active FIFO head, and a third submission blocks before its graph callback.
  • Carry {slot_id, generation} through TaskSlot and the local chip mailbox into the B1 runtime slot.
  • Publish direct-chip pipeline depth at startup and conservatively fall back to depth one for unsupported backends.
  • Reclaim graph-construction failures, queue partitions, reservations, and leases without dispatching unstarted device work.

Rebase and compatibility correction

  • Rebuilt B2 as one commit on main@8e89f014; old Feat: add launch-acceptance flight fence #1467/B1 stack commits are no longer present.
  • Permit the FIFO head to execute while its graph callback is still open. This preserves existing callbacks that submit device work and wait for L3-L2 communication before returning, while successors remain gated until every prior run is terminal.
  • Requeue a task blocked by a busy SUB worker into its original active-run partition instead of INVALID_RUN_ID.
  • Preserve deterministic run insertion order for legacy unscoped ready-queue access.
  • Address the changed-file lint and test-resource cleanup findings.

Validation

  • Changed-file pre-commit hooks passed: headers, English-only, large files, EOF/whitespace, clang-format, clang-tidy, cpplint, markdownlint, Ruff, and Pyright.
  • Full Python non-hardware UT on the code-identical pre-final-rebase tree: 866 passed, 10 deselected.
  • Final rebased commit 17d945bd: focused Python worker/L3-L2 UT 184 passed.
  • Full C++ non-hardware suite: 64/64 passed.
  • A2/A3 queued architecture probe: task_20260728_235654_212932915886, exit 0.
  • Final A2/A3 hardware task: task_20260729_000230_235780427710, exit 0.
  • Hardware cases passed: B2 whole-run FIFO, l3_l2_message_queue, and l3_l2_orch_comm_stream.
  • No local simulation run; GitHub simulation jobs remain automatic regression coverage.

Non-goals

This PR establishes bounded whole-run admission and FIFO dispatch. The two-frame endpoint, HBG prepared epochs, and TMR prepared state remain B3, B4, and B5.

@coderabbitai

coderabbitai Bot commented Jul 27, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The change introduces generation-safe pipeline leases, run-scoped FIFO admission, task-acceptance signaling, lease-aware mailbox dispatch, and AICore image-aware stream reuse across C++, Python bindings, device runners, documentation, and tests.

Changes

Pipeline admission and acceptance

Layer / File(s) Summary
Lease, contract, and callable identity contracts
src/common/worker/*, src/common/hierarchical/types.*, src/common/task_interface/*, src/common/platform/onboard/host/*
Pipeline leases, run-scoped queues, contract validation, callable AICore image hashes, arena selection, and runtime C-API entry points are added.
Run admission and scheduling
src/common/hierarchical/orchestrator.*, src/common/hierarchical/scheduler.*
Runs acquire generation-safe leases, track pending acceptances, activate through FIFO ordering, cancel unstarted work on failure, and dispatch only tasks belonging to the active run.
Mailbox and Python integration
src/common/hierarchical/worker_manager.*, python/bindings/*, python/simpler/*, docs/*
Mailbox payloads carry leases, TASK_ACCEPTED is observed before completion, acceptance callbacks update orchestration, and Python wrappers expose acceptance and lease-aware execution.
Device execution and stream reuse
src/common/worker/*, src/a2a3/*, src/common/platform/onboard/host/device_runner_base.*
Device execution selects leased slots and arena banks, validates generations, publishes acceptance after kernel enqueue, and recreates AICore streams when image hashes change.
Validation
tests/st/*, tests/ut/*
Tests cover lease generations, pipeline contracts, acceptance ordering, FIFO admission, stream reuse, callable identity, and concurrent waiters.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Poem

A rabbit hops through slots of two,
With leases fresh and generations new.
“Accepted!” the mailbox sings,
While streams remember AICore wings.
FIFO runs now safely flow—
Carrots queued in order, ho ho!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 18.35% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly matches the main change: bounded whole-run FIFO admission.
Description check ✅ Passed The description is directly aligned with the changeset and summarizes the same admission and FIFO dispatch work.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 9

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
src/common/hierarchical/orchestrator.cpp (1)

177-206: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Avoid racing fail_run_submission against scheduler activation. The run phase is sampled without the completion/run lock, and cancel_unstarted_run is later allowed to mark slots PENDING/READY/FREE as FAILED; if the closed run has already been activated to EXECUTING, the scheduler can dispatch the same slot as RUNNING concurrently, losing one terminal transition. Guard cancellation by the current run phase and use stateful atomic transitions for slot states.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/hierarchical/orchestrator.cpp` around lines 177 - 206, Update
fail_run_submission and cancel_unstarted_run to synchronize phase inspection
with the run/completion state, preventing cancellation from racing scheduler
activation. Recheck the current phase while holding the appropriate lock before
cancelling, and make each slot-state update use compare-and-exchange transitions
that only convert valid non-running states; never mark an EXECUTING/RUNNING slot
FAILED or overwrite a concurrent scheduler transition.
🧹 Nitpick comments (5)
src/common/platform/onboard/host/device_runner_base.h (1)

297-320: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Doc block missing the new aicore_image_hash param. The @param list enumerates every other argument; add a line for the new one so the staging contract stays self-describing (same for record_host_orch_callable at Line 332).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/platform/onboard/host/device_runner_base.h` around lines 297 -
320, Update the documentation for record_device_orch_callable to add an `@param`
entry describing aicore_image_hash, matching its position and purpose in the
function signature. Also add the corresponding parameter documentation to
record_host_orch_callable, without changing implementation behavior.
src/common/hierarchical/scheduler.cpp (1)

366-369: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Optional: fold the active_run_cb ? scoped : unscoped branching into helpers. The same "resolve active run, bail on invalid, pick scoped vs unscoped queue op" shape is now repeated in all four dispatch paths plus the wait predicate; a pair of small private helpers (e.g. resolve_active_run() returning std::optional<RunId> and pop_ready(...)) would keep future queue-API changes in one place.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/hierarchical/scheduler.cpp` around lines 366 - 369, Consolidate
the repeated active-run resolution and scoped/unscoped ready-queue branching
across all four dispatch paths and the wait predicate in the scheduler class.
Add small private helpers such as resolve_active_run() and pop_ready(...) that
preserve invalid-run early returns while centralizing queue API selection, then
update the affected paths to use them.
python/simpler/task_interface.py (1)

1277-1284: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Add one-line docstrings to match the neighbouring properties.

Every other property in this class documents its unit/semantics; pipeline_depth and runtime_slot_count are the odd ones out.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@python/simpler/task_interface.py` around lines 1277 - 1284, Add concise
one-line docstrings to the pipeline_depth and runtime_slot_count properties in
the task interface, matching the neighboring properties’ style and documenting
each value’s unit or semantics.
tests/ut/cpp/hierarchical/test_orchestrator.cpp (1)

656-665: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

A failed ASSERT_* between here and third.get() turns the test into a hang.

ASSERT_TRUE returns from the test body, and ~future from std::async blocks until the task completes — but begin_run() only returns once a slot frees, which is exactly what the skipped lines would have done. Consider releasing the admission slot from a scope guard so an assertion failure fails fast instead of wedging the suite.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/ut/cpp/hierarchical/test_orchestrator.cpp` around lines 656 - 665,
Ensure the asynchronous third begin_run test always releases an admission slot
before any ASSERT_* can exit the test, by adding a scope guard around the first
run’s slot ownership and consumption cleanup. Preserve the existing
successful-path behavior while making cleanup execute during assertion-induced
early returns, so third.get() cannot leave the test hanging.
src/common/platform/onboard/host/device_runner_base.cpp (1)

143-150: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick win

Bank validation accepts < PTO_PIPELINE_MAX_DEPTH but only two banks exist.

select_arena_bank admits any id below PTO_PIPELINE_MAX_DEPTH, while every consumer is a binary bank == 0 ? bank0 : bank1. With today's depth of 2 this is benign, but raising the depth constant would silently alias banks 2..N onto bank1 and share arenas across concurrently admitted runs. Prefer indexing an array of banks, or validate against the actual bank count.

♻️ Sketch
-    if (bank_id >= PTO_PIPELINE_MAX_DEPTH) {
-        LOG_ERROR("arena bank %u is outside [0, %u)", bank_id, PTO_PIPELINE_MAX_DEPTH);
+    if (bank_id >= kArenaBankCount) {
+        LOG_ERROR("arena bank %u is outside [0, %u)", bank_id, kArenaBankCount);
         return -1;
     }

with gm_heap_arena_[kArenaBankCount] etc. so the selection is gm_heap_arena_[g_arena_bank].

Also applies to: 193-211

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/common/platform/onboard/host/device_runner_base.cpp` around lines 143 -
150, Update select_arena_bank and the associated arena consumers to validate
against the actual number of supported banks rather than PTO_PIPELINE_MAX_DEPTH.
Define or reuse a two-bank count, reject IDs outside that range, and replace
binary bank-selection logic with indexed access so each accepted bank maps to
its own arena without aliasing.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/orchestrator.md`:
- Around line 101-106: The documentation around the “later submit” behavior must
distinguish admission from acceptance: clarify that acceptance unblocks only an
already-admitted successor, while a third submission with bounded depth two
remains blocked in begin_run() until a terminal run releases its lease. Preserve
the existing endpoint acceptance and completion fallback details.

In `@docs/worker-manager.md`:
- Around line 175-181: Update the documentation around the mailbox polling loop
to state that on_accept() is a one-shot callback, invoked only on the first
observation of TASK_ACCEPTED. Document the acceptance_observed guard and
preserve TASK_DONE as the mailbox ownership boundary, so implementations do not
repeatedly decrement per-run acceptance accounting.

In `@python/simpler/worker.py`:
- Line 1544: In the comment near the worker implementation, replace the
ambiguous multiplication symbol “×” in “N×40B” with the ASCII character “x”,
preserving the rest of the comment unchanged.
- Around line 180-186: Update _PIPELINE_LEASE_FMT to use the C ABI’s packed
12-byte layout for PipelineSlotLease: uint32 slot_id, uint32 reserved, and
uint64 generation with no padding between fields. Ensure _OFF_ARGS resolves to
the established MAILBOX_OFF_ARGS offset and remains compatible with the C++
definition in pto_runtime_c_api.h.

In `@src/a2a3/platform/onboard/host/device_runner.cpp`:
- Around line 621-624: Make acceptance durable across the mailbox transition in
the dispatch flow around publish_task_accepted and
LocalMailboxEndpoint::run_with_accept: ensure a parent that observes TASK_DONE
without previously sampling TASK_ACCEPTED still invokes on_accept exactly once.
Use a latched acceptance flag/counter or equivalent state, and preserve the
existing behavior for parents that observe TASK_ACCEPTED first so
decrement_run_accepts is not duplicated.

In `@src/common/hierarchical/orchestrator.cpp`:
- Around line 360-371: Update Orchestrator::decrement_run_accepts to acquire
run->completion_mu before changing pending_accepts and before calling
completion_cv.notify_all(). Keep the decrement, underflow handling, error
recording, and notification behavior intact while ensuring the predicate
transition is synchronized with wait_run_accepted.

In `@src/common/hierarchical/types.cpp`:
- Around line 70-82: Update ReadyQueue’s unscoped access paths, including
try_pop(TaskSlot&) and try_front(TaskSlot&), so they no longer select buckets by
unordered_map iteration order; remove these overloads or add separate
run-insertion-order tracking that preserves FIFO, and update callers such as
try_pop_group(TaskSlot&) and try_pop_single(worker_id, TaskSlot&) accordingly.

In `@src/common/hierarchical/worker_manager.cpp`:
- Around line 397-404: Move the acceptance callback out of the mailbox-locked
section in the worker task flow around read_mailbox_state and
LocalMailboxEndpoint::run. Record that TASK_ACCEPTED was observed while polling,
then invoke on_accept_ only after the mailbox round-trip and mailbox_mu_ have
been released, preserving exactly-once callback behavior.

In `@tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py`:
- Line 114: After unpacking tensors into first_a, first_b, first_out, second_a,
second_b, and second_out, explicitly delete those references before
free_host_buffer runs; apply the same cleanup at the corresponding later
unpacking block, while guarding it for early-failure paths where the names may
not be bound.

---

Outside diff comments:
In `@src/common/hierarchical/orchestrator.cpp`:
- Around line 177-206: Update fail_run_submission and cancel_unstarted_run to
synchronize phase inspection with the run/completion state, preventing
cancellation from racing scheduler activation. Recheck the current phase while
holding the appropriate lock before cancelling, and make each slot-state update
use compare-and-exchange transitions that only convert valid non-running states;
never mark an EXECUTING/RUNNING slot FAILED or overwrite a concurrent scheduler
transition.

---

Nitpick comments:
In `@python/simpler/task_interface.py`:
- Around line 1277-1284: Add concise one-line docstrings to the pipeline_depth
and runtime_slot_count properties in the task interface, matching the
neighboring properties’ style and documenting each value’s unit or semantics.

In `@src/common/hierarchical/scheduler.cpp`:
- Around line 366-369: Consolidate the repeated active-run resolution and
scoped/unscoped ready-queue branching across all four dispatch paths and the
wait predicate in the scheduler class. Add small private helpers such as
resolve_active_run() and pop_ready(...) that preserve invalid-run early returns
while centralizing queue API selection, then update the affected paths to use
them.

In `@src/common/platform/onboard/host/device_runner_base.cpp`:
- Around line 143-150: Update select_arena_bank and the associated arena
consumers to validate against the actual number of supported banks rather than
PTO_PIPELINE_MAX_DEPTH. Define or reuse a two-bank count, reject IDs outside
that range, and replace binary bank-selection logic with indexed access so each
accepted bank maps to its own arena without aliasing.

In `@src/common/platform/onboard/host/device_runner_base.h`:
- Around line 297-320: Update the documentation for record_device_orch_callable
to add an `@param` entry describing aicore_image_hash, matching its position and
purpose in the function signature. Also add the corresponding parameter
documentation to record_host_orch_callable, without changing implementation
behavior.

In `@tests/ut/cpp/hierarchical/test_orchestrator.cpp`:
- Around line 656-665: Ensure the asynchronous third begin_run test always
releases an admission slot before any ASSERT_* can exit the test, by adding a
scope guard around the first run’s slot ownership and consumption cleanup.
Preserve the existing successful-path behavior while making cleanup execute
during assertion-induced early returns, so third.get() cannot leave the test
hanging.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 976e895c-87d2-41d9-ab42-f455679ef0bc

📥 Commits

Reviewing files that changed from the base of the PR and between f44c715 and ae22766.

📒 Files selected for processing (45)
  • docs/orchestrator.md
  • docs/task-flow.md
  • docs/worker-manager.md
  • python/bindings/task_interface.cpp
  • python/bindings/worker_bind.h
  • python/simpler/orchestrator.py
  • python/simpler/task_interface.py
  • python/simpler/worker.py
  • src/a2a3/platform/onboard/host/device_runner.cpp
  • src/a2a3/platform/onboard/host/device_runner.h
  • src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp
  • src/a2a3/runtime/tensormap_and_ringbuffer/host/runtime_maker.cpp
  • src/a5/runtime/host_build_graph/host/runtime_maker.cpp
  • src/a5/runtime/tensormap_and_ringbuffer/host/runtime_maker.cpp
  • src/common/hierarchical/orchestrator.cpp
  • src/common/hierarchical/orchestrator.h
  • src/common/hierarchical/scheduler.cpp
  • src/common/hierarchical/scheduler.h
  • src/common/hierarchical/types.cpp
  • src/common/hierarchical/types.h
  • src/common/hierarchical/worker.cpp
  • src/common/hierarchical/worker.h
  • src/common/hierarchical/worker_manager.cpp
  • src/common/hierarchical/worker_manager.h
  • src/common/platform/onboard/host/c_api_shared.cpp
  • src/common/platform/onboard/host/device_runner_base.cpp
  • src/common/platform/onboard/host/device_runner_base.h
  • src/common/platform/sim/host/c_api_shared.cpp
  • src/common/task_interface/chip_callable_layout.h
  • src/common/task_interface/prepare_callable_common.h
  • src/common/utils/fnv1a_64.h
  • src/common/worker/chip_worker.cpp
  • src/common/worker/chip_worker.h
  • src/common/worker/pipeline_contract.h
  • src/common/worker/pipeline_slot_pool.h
  • src/common/worker/pto_runtime_c_api.h
  • tests/st/a2a3/host_build_graph/run_stream_reuse/kernels/aiv/kernel_sub.cpp
  • tests/st/a2a3/host_build_graph/run_stream_reuse/test_run_stream_reuse.py
  • tests/st/a2a3/host_build_graph/worker_async_fifo/test_worker_async_fifo.py
  • tests/ut/cpp/hierarchical/test_orchestrator.cpp
  • tests/ut/cpp/hierarchical/test_pipeline_contract.cpp
  • tests/ut/cpp/hierarchical/test_scheduler.cpp
  • tests/ut/cpp/types/test_chip_callable_upload_immutable.cpp
  • tests/ut/py/test_callable_identity.py
  • tests/ut/py/test_worker/test_host_worker.py

Comment thread docs/orchestrator.md
Comment thread docs/worker-manager.md Outdated
Comment thread python/simpler/worker.py
Comment thread python/simpler/worker.py Outdated
Comment thread src/a2a3/platform/onboard/host/device_runner.cpp
Comment thread src/common/hierarchical/orchestrator.cpp
Comment thread src/common/hierarchical/types.cpp Outdated
Comment thread src/common/hierarchical/worker_manager.cpp
@Crane-Liu
Crane-Liu force-pushed the codex/worker-async-b2-whole-run-fifo branch from ae22766 to 8fcabcf Compare July 29, 2026 06:59
@Crane-Liu
Crane-Liu force-pushed the codex/worker-async-b2-whole-run-fifo branch from 8fcabcf to 17d945b Compare July 29, 2026 07:05
@Crane-Liu

Copy link
Copy Markdown
Contributor Author

B2 was rebuilt and force-with-lease updated as a single commit on the current main.

Key correction: the FIFO head can execute while its graph callback is still open, so existing L3-L2 interactive callbacks no longer deadlock. A successor still cannot dispatch until the previous run is terminal. The busy-SUB requeue path now preserves the active run partition, and unscoped queue access preserves run insertion order.

Validation is recorded in the PR body. Final hardware task task_20260729_000230_235780427710 passed the B2 FIFO case and both previously timing-out L3-L2 examples on A2/A3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant